On Uniformly Sampling Traces of a Transition System (Extended Version)

A key problem in constrained random verification (CRV) concerns generation of input stimuli that result in good coverage of the system's runs in targeted corners of its behavior space. Existing CRV solutions however provide no formal guarantees on the distribution of the system's runs. In this paper, we take a first step towards solving this problem. We present an algorithm based on Algebraic Decision Diagrams for sampling bounded traces (i.e. sequences of states) of a sequential circuit with provable uniformity (or bias) guarantees, while satisfying given constraints. We have implemented our algorithm in a tool called TraceSampler. Extensive experiments show that TraceSampler outperforms alternative approaches that provide similar uniformity guarantees.Comment: Extended version of paper that will appear in proceedings of International Conference on Computer-Aided Design (ICCAD '20); changed wrong text color in sec 7; added 'extended version

Design of Experiments to Evaluate CAD Algorithms: Which Improvements Are Due to Improved Heuristic and Which Are Merely Due to Chance?

. More than a thousand mathematical problems arising in engineering and science have been shown to be NP-hard. Problems of practical size that are NP-hard can only be solved by devising polynomial-time heuristics, with no guarantee whatsoever on the quality of the solution. Extensive experimentation and comparative analysis is required before we can decide on a `better heuristic'. Past efforts to do either have been, for the most part, ad hoc. While there is no shortage of published claims of `incremental improvements' with a particular heuristic, they are not supported by a test of hypothesis such as `Is the improvement due to improved heuristic used by the algorithm or due merely to chance?' This paper introduces motivation, context, and an approach to begin addressing and resolving such hypotheses using the design of experiment techniques rooted in the scientific method. We accept the methods of experimental design: randomization, replication, and organization to reduce error, first..

A Brief Tour of the Home Page on WWW Statistical Experiment Archives: Benchmark Descriptions and Posted Solutions to NP-hard Problems

More than a thousand mathematical problems arising in engineering and science have been shown to be NP-hard. Problems of practical size that are NP-hard can only be solved by devising polynomial-time heuristics, with no guarantee whatsoever on the quality of the solution. Extensive experimentation and comparative analysis is required before we can decide on a `better heuristic'. Web archives on this site support the design of experiments to evaluate performance of algorithms that solve NP-hard problem. The purpose of this report is to give the reader a brief navigational tour of the current directories, address the challenges goals and objectives of these archives, contrast current approaches to the design and reporting of benchmarking experiments with alternatives that are based on proven principles from statistics, provide a number of illustrative, tutorial-level examples, and motivate the reader to participate in collaborative, peer-reviewed web-based benchmarking experiments duri..